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Abstract 

This paper considers energy-aware control for a computing system with two states: active and idle. In the active 
^ state, the controller chooses to perform a single task using one of multiple task processing modes. The controller then 

I saves energy by choosing an amount of time for the system to be idle. These decisions affect processing time, energy 

expenditure, and an abstract attribute vector that can be used to model other criteria of interest (such as processing 
CsJ quality or distortion). The goal is to optimize time average system performance. Applications of this model include 

Q a smart phone that makes energy-efficient computation and transmission decisions, a computer that processes tasks 

O subject to rate, quality, and power constraints, and a smart grid energy manager that allocates resources in reaction 

to a time varying energy price. The solution methodology of this paper uses the theory of optimization for renewal 
systems developed in our previous work. This paper is written in tutorial form and develops the main concepts of the 
theory using several detailed examples. It also highlights the relationship between online dynamic optimization and 
linear fractional programming. Finally, it provides exercises to help the reader learn the main concepts and apply 
them to their own optimizations. This paper is an arxiv technical report, and is a preliminary version of material 
that will appear as a book chapter in an upcoming book on green communications and networking. 
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I. Introduction 

^ This paper considers energy-aware control for a computing system with two states: active and idle. In the active 

C ■ state, the controller chooses to perform a single task using one of multiple task processing modes. The controller 

^ then saves energy by choosing an amount of time for the system to be idle. These decisions affect processing 

^ time, energy expenditure, and an abstract attribute vector that can be used to model other criteria of interest (such 
as processing quality or distortion). The goal is to optimize time average system performance. Applications of 

T-H this model include a smart phone that makes energy-efficient computation and transmission decisions, a computer 

^ that processes tasks subject to rate, quality, and power constraints, and a smart grid energy manager that allocates 

• . resources in reaction to a time varying energy price. 

• ^ The solution methodology of this paper uses the theory of optimization for renewal systems developed in [1][2]. 

^ Section [n] focuses on a computer system that seeks to minimize average power subject to processing rate constraints 

^ — — 



for different classes of tasks. Section III generalizes to treat optimization for a larger class of systems. Section IV 
extends the model to allow control actions to react to a random event observed at the beginning of each active 
period, such as a vector of current channel conditions or energy prices. 

II. Task Scheduling with Processing Rate Constraints 

To illustrate the method, this section considers a particular system. Consider a computer system that repeatedly 
processes tasks. There are N classes of tasks, where is a positive integer. For simplicity, assume each class 
always has a new task ready to be performed (this is extended to randomly arriving tasks in Section III-C| ). The 



system operates over time intervals called frames. Each frame k G {0, 1,2, . . .} begins with an active period of 
size D[k] and ends with an idle period of size I[k] (see Fig. [T]). At the beginning of each active period k, the 
controller selects a new task of class c[k] G {!,..., N}. It also chooses a processing mode m[k] from a finite set 
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Fig. 1. (a) A processor that chooses from one of A'^ classes on each frame k. (b) A timeline illustrating the active and idle periods for each 
frame. 



M. of possible processing options. These control decisions affect the duration of the active period and the 
energy e[A;] that is incurred. The controller then selects an amount of time /[fc] to be idle, where is chosen 
such that < /[fc] < Imax for some positive number Imax- Choosing /[fc] =0 effectively skips the idle period, 
so there can be back-to-back active periods. For simplicity, this section assumes that no energy is expended in the 
idle state. 

Assume that D\k\ and are random functions of the class and mode decisions for frame k. Specifically, 
assume D\k\ and e[A;] are conditionally independent of the past, given the current (c[A;], m[A;]) that is used, with 
mean values given by functions l)(c, m) and e(c, m) defined over (c, m) G {1, . . . , A^} x M.: 

b{c,m.)^^\D\k\\{c\k\,m\k\) = (c,m)] , e{c,m)AE[e[k]\{c[k],m[k]) = (c,m)] 

where the notation "aAb" represents "a is defined to be equal to b." This paper uses D{c[k], m[k]) and e{c[k],m[k]) 
to denote expectations given a particular decision (c[/i;], for frame k: 

D{c[k],m[k]) = E[D[k]\c[k],m[k]] , e{c[k],m[k]) = E[e[k]\c[k],m[k]] 

It is assumed there is a positive value Dmin such that D[k] > Dmin for all frames k, regardless of the (c[fc],m[fc]) 
decisions. Thus, all frame sizes are at least Dmin units of time. Further, for technical reasons, it is assumed that 
second moments of D[k] and e[k] are bounded by a finite constant a^, so that: 



E [D[kf] < , E [e[kf] < 



(1) 



where ([T]l holds regardless of the policy for selecting (c[A;], The conditional joint distribution of (-D[A;], e[k]), 

given (c[A;], is otherwise arbitrary, and only the mean values D{c,m) and e(c, m) are known for each 
c G {1, . . . and m G M.. In the special case of a deterministic system, the functions D{c,m) and e{c,m) 
can be viewed as deterministic mappings from a given control action (c[A;], m[A;]) = (c, m) to the actual delay 
D[k] = D{c,m) and energy e[k] = e(c, m) experienced on frame k, rather than as expectations of these values. 



A. Examples of Energy-Aware Processing 

Consider an example where a computer system performs computation for different tasks. Suppose the system 
uses a chip multi-processor that can select between one of multiple processing modes for each task. For example, 
this might be done using voltage/frequency scaling, or by using a choice of different processing cores [3] [4]. Let 
M. represent the set of processing modes. For each mode m G 7W, define: 

Tsetup{fn) = Setup time for mode m. 

Gsetupim) = Setup energy for mode m. 

DPI{m) = Average delay-per-instruction for mode m. 

EPI{m) = Average energy-per-instruction for mode m. 

Further suppose that tasks of class c G {!,..., A^} have an average number of instructions equal to Sc- For 
simplicity, suppose the number of instructions in a task is independent of the energy and delay of each individual 
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instruction. Then the average energy and delay functions e(c, m) and D{c, m) are: 

e{c,m) = esetupim) + ScEPI{m) 
D{c,m) = Tsetupim) + ScDPI{m) 

In cases when the system cannot be modeled using DPI{m) and EPI{m) functions, the expectations e(c, m) and 
D{c,m) can be estimated as empirical averages of energy and delay observed when processing type c tasks with 
mode m. While this example assumes all classes have the same set of processing mode options Ai, this can easily 
be extended to restrict each class c to its own subset of options M-c- 

As another example, consider the problem of wireless data transmission. Here, each task represents a packet of 
data that must be transmitted. Let Ai represent the set of wireless transmission options (such as modulation and 
coding strategies). For each m G A4, define fi{m) as the transmission rate (in bits per unit time) under option m, 
and let power{m) be the power used. For simplicity, assume there are no transmission errors. Let Be represent the 
average packet size for class c G {1, . . . , N}, in units of bits. Thus: 

e{c,m) = power{m)Bc/ ii{m) 
D{c,m) = Bc/fJ-{m) 

In the case when each transmission mode m £ A4 has a known error probability, the functions e(c, m) and Z)(c, m) 
can be redefined to account for retransmissions. An important alternative scenario is when channel states are time- 
varying but can be measured at the beginning of each frame. This can be treated using the extended theory in 
Section |lVl 



B. Time Averages as Ratios of Frame Averages 

The goal is to design a control policy that makes decisions over frames to minimize time average power subject 
to processing each class n G {1, . . . , A^} with rate at least A^, for some desired processing rates (Ai, . . . , Aat) that 
are given. Before formalizing this as a mathematical optimization, this subsection shows how to write time averages 
in terms of frame averages. Suppose there is a control policy that yields a sequence of energies {e[0], e[l], e[2], . . .} 
and corresponding frame sizes {D[0] + I[0],D[1] + I[1],D[2] + I[2], . . .} for each frame k £ {0, 1, 2, . . .}. The 
frame averages e, D, I are defined: 

K-l K-1 

eA lim - e[k] , DA lim - J] D[k] , lA lim - J] I[k] (2) 

k=0 k=0 k=0 

where, for simplicity, it is assumed the limits converge to constants with probability 1. Note that e does not represent 
the time average power used by the system, because it does not consider the amount of time spent in each frame. 
The time average power considers the accumulated energy used divided by the total time, and is written as follows: 



"■^^ eSi wi + 'W) "^''■kEk^aiom + m) D + i 

Therefore, the time average power is equal to the average energy per frame divided by the average frame size. This 
simple observation is often used in renewal-reward theory [5] [6]. 

For each class n G {!,..., N} and each frame k, define an indicator variable ln[k] that is 1 if the controller 
chooses to process a class n task on frame k, and else: 

1 if c[k] = n 
if c[k] / n 



ln[k]-- 



Then ln=Yim.K^oo Yl,k=o ^n[k] is the fraction of frames that choose class n, and the ratio ln/{D + /) is the 
time average rate of processing class n tasks, in tasks per unit time. 
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The problem of minimizing time average power subject to processing each class n at a rate of at least A„ tasks 
per unit time is then mathematically written as follows: 

e 

Minimize: — — - (3) 
D + I 

Subject to: > A„ Vn G {1, . . . , A^} (4) 

{c[k],m[k])£{l,...,N}xM VA; E {0,1,2,...} (5) 
0<I[k]<Imax VA; G {0,1,2,...} (6) 

where the objective (|3]) is average power, the constraint Q ensures the processing rate of each class n is at least 
Xn, and constraints ([Sf-Q ensure that c[k] S {1, . . . , A^}, m[k] G A4, and < I[k] < Imax for each frame k. 



C. Relation to Frame Average Expectations 

The problem ([3])-(|6]l is defined by frame averages. This subsection shows that frame averages are related to 
frame average expectations, and hence can be related to the expectation functions D{c,m), e{c,m). Consider any 
(possibly randomized) control algorithm for selecting (c[/i:], m[A;]) over frames, and assume this gives rise to well 
defined expectations E [e[k]] for each frame k. By the law of iterated expectations, it follows that for any given 
frame k G {0,1,2,...}: 

E[e[A;]] = E[E[e[k]\c[k],m[k]]] =E[e(c[k],m[k])] (7) 

Furthermore, because second moments are bounded by a constant on each frame k, the bounded moment 
convergence theorem (given in Appendix A) ensures that if the frame average energy converges to a constant e 
with probability 1, as defined in (|2]), then e is the same as the frame average expectation: 

^ K-l ^ K~l 

lim — > e\k] =e= lim — > E [e(c[/i;l, m[/cl)l 
K^oo K K^oo K ^ 

k=0 k=0 

The same goes for the quantities D, I, 1„. Hence, one can interpret the problem (|3]l-([6]) using frame average 
expectations, rather than pure frame averages. 



D. An Example with One Task Class 

Consider a simple example with only one type of task. The system processes a new task of this type at the 
beginning of every busy period. The energy and delay functions can then be written purely in terms of the processing 
mode m G M., so that we have e{m) and D{m). Suppose there are only two processing mode options, so that 
A4 = {1,2}, and that each option leads to a deterministic energy and delay, as given below: 

m[k] = l =^ {e[k],D[k]) = {e{l),D{l)) = {l,7) (8) 
m[k]=2 =^ {e[k],D[k]) = {e{2),D{2)) = {3A) (9) 

Option m[k] = I requires 1 unit of energy but 7 units of processing time. Option m[k] = 2 is more energy-expensive 
(requiring 3 units of energy) but is faster (taking only 4 time units). The idle time I[k] is chosen every frame in 
the interval [0, 10], so that I-max = 10. 

1) No constraints: For this system, suppose we seek to minimize average power e/{D + I), with no processing 
rate constraint. Consider three possible algorithms: 

1) Always use m[k] = 1 and I[k] = I for all frames k, for some constant I G [0, 10]. 

2) Always use m[k] = 2 and I[k] = I for all frames k, for some constant / G [0, 10]. 

3) Always use I[k] = / for all frames k, for some constant I G [0, 10]. However, each frame k, independently 
choose m[k] = \ with probability p, and m[k] = 2 with probability 1—p (for some p that satisfies < p < 1). 
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Clearly the third algorithm contains the first two for p = 1 and p = 0, respectively. The time average power 
under each algorithm is: 

e e(l) 1 



m[k] = 1 always 
m[k] =2 always 
probabilistic rule 



D + I D{l) + I 7 + 1 

e _ e(2) _ 3 
D + I ~ b{2) + I ~ 4 + 1 

e pe{l) + {I - p)e{2) lp + 3(l-p) 



D + I pb{l) + {l-p)b{2) + I 7p + 4(l-p) + / 

It is clear that in all three cases, we should choose / = 10 to minimize average power. Further, it is clear that 
always choosing m[k] = 1 is better than always choosing m[k] = 2. However, it is not immediately obvious if a 
randomized mode selection rule can do even better. The answer is no: In this case, power is minimized by choosing 
m[k] = 1 and I[k] = 10 for all frames k, yielding average power 1/17. 

This fact holds more generally: Let a(a) and b{a) be any deterministic real valued functions defined over general 
actions a that are chosen in an abstract action space A. Assume the functions are bounded, and that there is a 
value bmin > such that 6(a) > bmin for all a ^ A. Consider designing a randomized choice of a that minimizes 
E [a{a)] /E [6(a)], where the expectations are with respect to the randomness in the a selection. The next lemma 
shows this is done by deterministically choosing an action a ^ A that minimizes a(a)/6(a). 

Lemma 1: Under the assumptions of the preceding paragraph, for any randomized choice of a G ^ we have: 

a(a) 



E[a(a)]^ .^^ 



6(a) 



E [6(a)] - aeA 

and so the infimum of the ratio over the class of deterministic decisions yields a value that is less than or equal to 
that of any randomized selection. 

Proof: Consider any randomized policy that yields expectations E [a(a)] and E [6(a)]. Without loss of generality, 
assume these expectations are achieved by a policy that randomizes over a finite set of M actions ai , . . . , ajv/ in 
A with some probabilities pi, . . . ,pAf Q 

M M 

E [a(a)] = ^ Pma{am) , E [6(a)] = ^ Prah{am) 

m=l m=l 

Then, because 6(am) > for all Um, we have: 

IE [«(«)] _ Em=l^'m«(«m) 



y Em=iPrnfc(«m) infQ6^[a(a)/6(a)] 
= inf [a(a)/6(a)] 

□ 

2 ) One Constraint: The preceding subsection shows that unconstrained problems can be solved by deterministic 
actions. This is not true for constrained problems. This subsection shows that adding just a single constraint often 
necessitates the use of randomized actions. Consider the same problem as before, with two choices for m[k], and 
with the same e{m) and D{m) values given in ([8])-(|9]l. We want to minimize e/{D + I) subject to 1/{D + I) > 1/5, 
where 1/{D + I) is the rate of processing jobs. The constraint is equivalent to D + I < 5. 



'indeed, because the set 5 = {{a{a), b{a)) such that a G A} is bounded, the expectation (E [a(a)] , E [&(«)]) is finite and is contained 
in the convex hull of 5. Thus, (E [a(Q)] ,E [b(a)]) is a convex combination of a finite number of points in 5. 
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Assume the algorithm chooses I[k\ over frames to yield an average / that is somewhere in the interval < / < 10. 
Now consider different algorithms for selecting m[k]. If we choose m[k] = 1 always (so that (e(l), £)(!)) = (1, 7)), 
then: 

e 1 — - - 

m\k] = 1 always = — = = = , D + I = 7 + I 

D+I 7+1 

and so it is impossible to meet the constraint D + 1 < 5, because 7 + / > 5. 
If we choose m[k] = 2 always (so that {e{2),D(2)) = (3,4)), then: 

g 3 _ _ 

m\k] = 2 always =^ = — = = = , D + I = A + I 

^ ^ D+I 4+1 

It is clear that we can meet the constraint by choosing / so that < / < 1, and power is minimized in this setting 
by using 1 = 1. This can be achieved, for example, by using I[k] = 1 for all frames k. This meets the processing 
rate constraint with equality: D + I = 4 + l = 5. Further, it yields average power e/{D + 1) = 3/5 = 0.6. 

However, it is possible to reduce average power while also meeting the constraint with equality by using the 
following randomized policy (which can be shown to be optimal): Choose I[k] = for all frames k, so that / = 0. 
Then every frame k, independently choose m[k] = 1 with probability 1/3, and m[k] = 2 with probability 2/3. We 
then have: 

D + 1 = (1/3)7 + (2/3)4 + = 5 
and so the processing rate constraint is met with equality. However, average power is: 

" = '/(f !?{f = 7/15 . 0.406667 

D + I (1/3)7 + (2/3)4 + ' 

This is a significant savings over the average power of 0.6 from the deterministic policy. 

E. The Linear Fractional Program for Task Scheduling 

Now consider the general problem ([Sjl-Q for minimizing average power subject to average processing rate 
constraints for each of the L classes. Assume the problem is feasible, so that it is possible to choose actions 
(c[A;], m[/c], /[A;]) over frames to meet the desired constraints Q-Q. It can be shown that an optimal solution 
can be achieved over the class of stationary and randomized policies with the following structure: Every frame, 
independently choose vector (c[fc],m[/c]) with some probabilities p{c,m) = Pr[{c[k],m[k]) = (c, m)]. Further, use 
a constant idle time I[k] = I for all frames k, for some constant / that satisfies < / < Imax- Thus, the problem 
can be written as the following linear fractional program with unknowns p{c, m) and / and known constants 
e{c,m), D{c,m), \n, and /^ax: 

E^i Emg^i Pjc, m)eic, m) 

p{c, m)D{c, m) 

I + Ef=i EmG>! rn)D{c, m) 

< I < Imax (12) 

p(c,m)>0 VcG {1,...,A^}, VmG (13) 

N 

J^J]p(c,m) = l (14) 

c=l m^M 



Minimize: , ^c=i^rn^MfK . / v^.^ j ^^q^ 

JC= 

Subject to: ""^^ > K Vn E {1, . . . , A^} (11) 



where the numerator and denominator in ( [TO] ) are equal to e and D + I, respectively, under this randomized 
algorithm, the numerator in the left-hand- side of ( [TT| ) is equal to 1„, and the constraints ([T3])-([l4]) specify that 
p{c, m) must be a valid probability mass function. 

Linear fractional programs can be solved in several ways. One method uses a nonlinear change of variables to 
map the problem to a convex program [7] . However, this method does not admit an online implementation, because 
time averages are not preserved through the nonlinear change of variables. Below, an online algorithm is presented 
that makes decisions every frame k. The algorithm is not a stationary and randomized algorithm as described above. 
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However, it yields time averages that satisfy the desired constraints of the problem ([Sjl-Q, with a time average 
power expenditure that can be pushed arbitrarily close to the optimal value. A significant advantage of this approach 
is that it extends to treat cases with random task arrivals, without requiring knowledge of the (Ai, . . . , X]\j) arrival 
rates, and to treat other problems with observed random events, without requiring knowledge of the probability 
distribution for these events. These extensions are shown in later sections. 

For later analysis, it is useful to write (^Oi-1 14) in a simpler form. Let power°P^ be the optimal time average 
power for the above linear fractional program, achieved by some probability distribution p*(c, m) and idle time 
/* that satisfies < /* < Imax- Let (c*[A;], m*[/c], /*[A;]) represent the frame k decisions under this stationary and 
randomized policy. Thus: 

E[e{c*[k],m*[k])] 



E 



I*[k] + D{c*[k],m*[k]) 



power 



opt 



E 



I*[k] + D{c*[k],m*[k]) 



> \n VnG {l,...,iV} 



(15) 



(16) 



where ln[k\ is an indicator function that is 1 if c*[k] = n, and else. The numerator and denominator of ( [15] ) 



correspond to those of ([TO]). Likewise, the constraint ( [T6[ ) corresponds to ([TTj)- 
F. Virtual Queues 



An(D[k]+l[k]) 





Qn[k] 









ln[l<] 



Fig. 2. An illustration of the virtual queue Q„[fc] from equation ([Tsl 



To solve the problem (|3|)-([6]), we first consider the constraints ([4|), which are equivalent to the constraints: 

A„(:D + 7)<T„ VnG{l,...,iV} (17) 
For each constraint n G {1, . . . , N}, define a virtual queue Qn[k] that is updated on frames k G {0, 1,2,.. .} by: 

Qn[k + 1] = max[Q„[fe] + \n{D[k\ + I[k]) - l„[fc], 0] (18) 

The initial condition Qn[^] can be any non-negative value. For simplicity, it is assumed throughout that Q„[0] = 
for all n G {1, . . . , N}. The update ( [T8[ ) can be viewed as a discrete time queueing equation, where Qn[k] is the 
backlog on frame k, \n{D[k\ + I[k\) is an effective amount of "new arrivals," and ln[k] is the amount of "offered 
service" (see Fig. |2]). The intuition is that if all virtual queues Qn[k] are stable, then the average "arrival rate" 
Xn{D + /) must be less than or equal to the average "service rate" 1„, which ensures the desired constraint ( 17 1. 
This is made precise in the following lemma. 

Lemma 2: (Virtual Queues) Suppose Qn[k] has update equation given by ( [T8[ ), with any non-negative initial 
condition. 

a) For all K G {1, 2, 3, . . .} we have: 



1 

Y,[\^{D[k]+I[k\) 



K 



ln[k]] < 



k=0 



Qn[K]-Qn[0] 

K 



(19) 



b) If limx^-oo Qn[K]/ K = with probability 1, then: 

K-1 



limsup ^ [\n{D[k] + I[k\) - ln[k\] < with probability 1 



(20) 



k=0 



8 



c) If limi^-^oo IE [Qn[K]] /K = 0, then: 

limsup[A„(D[i^] + 1[K]) - ln[K]] < (21) 

K^oo 

where D[K], 1[K], T„[K] are defined: 



Proof: From ([18) we have for all k G {0, 1,2,.. .}: 

Qn[k + 1] > + Xn{D[k] + /[A;]) - ln[k] 

Fixing a positive integer K and summing the above over k G {0, . . . , K — 1} yields: 



K-l 

Qn[K] - Qn[0] > J2 [^n{D[k] + I[k]) 
k=0 



Dividing the above by K proves part (a). Part (b) follows from ( [T9| ) by taking a lim sup. Part (c) follows by first 
taking expectations of ^T9\ and then taking a lim sup. □ 

Inequality ^T9\ shows that the value Qn[K]/K bounds the amount by which the desired constraint for class 
n is violated by the time averages achieved over the first K frames. Suppose that D[k], I[k\, and ln[k] have 
frame averages that converge to constants D, I, In with probability 1. Part (c) of Lemma |2] indicates that if 
limx^oo E [Qn[K]] /K = for all n G {1, . . . , N}, then A„,(Z) + J) < T„ for all n G {1, . . . , N}. 

In the language of queueing theory, a discrete time queue Q[k] is said to be rate stable if lim/s_i.oo Q[k]/k = 
with probability 1, and is mean rate stable if limfc_^oo IE [Q[k]] /k = [I]. With this terminology, the above lemma 



shows that if Qn[k] is rate stable then ([20]) holds, and if Qn[k] is mean rate stable then ([2T]) holds 



G. The Drift-Plus-Penalty Ratio 

To stabilize the queues while minimizing time average power, we use Lyapunov optimization theory, which gives 
rise to the drift-plus-penalty ratio algorithm [1]. First define L[k] as the sum of the squares of all queues on frame 
k (divided by 2 for convenience later): 

1 ^ 

L[k]^-Y.Q-[kf 

n=l 

L[k\ is often called a Lyapunov function, and acts as a scalar measure of the size of the queues. Intuitively, keeping 
L[k] small leads to stable queues, and we should take actions that tend to shrink L[k] from one frame to the next. 
Define A[A;] as the Lyapunov drift, being the difference in the Lyapunov function from one frame to the next: 

/\[k]^L[k + I] - L[k] 

Taking actions to minimize A [k] every frame can be shown to ensure the desired constraints are satisfied whenever 
it is possible to satisfy them, but does not incorporate power minimization. To incorporate this, every frame k 
we observe the current queue vector Q[k] = {Qi[k], . . . ,QN[k]) and choose control actions (c[fc], m[A:], /[A;]) to 
minimize a bound on the following drift-plus-penalty ratio: 

E[A[fc] + ye[fc]|Q[fc]] 
¥.[D[k]+I[k]\Q[k]] 

where y is a non-negative parameter that weights the extent to which power minimization is emphasized. The 
intuition is that the numerator incorporates both drift and energy. The denominator "normalizes" this by the expected 
frame size, with the understanding that average power must include both energy and frame size. We soon show 
that this intuition is correct, in that all desired time average constraints are satisfied, and the time average power 
is within 0{\/V) of the optimal value power^P^. Hence, average power can be pushed arbitrarily close to optimal 
by using a sufficiently large value of V. The tradeoff is that average queue sizes grow with V, which impacts the 
convergence time required to satisfy the desired constraints. 

The drift-plus-penalty ratio method was first developed for the context of restless bandit systems in [8] [9]. The 
method was used for optimization of renewal systems in [1][2], which treat problems similar to those considered 
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in this paper. In the special case when all frame sizes are fixed and equal to one unit of time (a time slot), and 
when V = 0, the method reduces to observing queues Q[k] every slot k and taking actions to minimize a bound 
on E [A[A;]|(5[A;]]. This is the rule that generates the classic max- weight scheduling algorithms for queue stability 
(without performance optimization), developed by Tassiulas and Ephremides in [10][11]. For systems with unit 
time slots but with V > 0, the drift-plus-penalty ratio technique reduces to the drift-plus-penalty technique of 
[12] [13] [14], which treats joint queue stability and penalty minimization in systems with unit size slots. 

1 ) Bounding the Drift-Plus-Penalty Ratio: To construct an explicit algorithm, we first bound the drift-plus-penalty 
ratio. 

Lemma 3: For all frames k £ {0, 1,2,.. .}, all possible Q[k], and under any decisions for (c[A;], I[k]), we 
have: 

E[A[k] + Ve[k]\Q[k]] ^ B , ¥.[Ve{c[klm[k])\Q[k\] 



E[D[k]+I[k]\Q[kl 



nm+m\QM e 



b{c[k],m[k]) + I[k]\Q[k] 



+ - 





\n{b{c[klm[k]) + I[k]) - 


-ln[k]\Q[k] 


E 


b{c[k],m[k]) + I[k]\Q[k] 





(22) 



where i? is a constant that satisfies the following for all possible Q[k] and all policies: 



1 ^ 

i? > - J^E [{\n{D[k] + I[k]) - Uk]f\Q[k]] 



n=l 



Such a constant B exists by the second moment boundedness assumptions ([T]). 
Proof: Note by iterated expectations that (similar to 



¥.[D[k\\Q[k]]=¥. b{c[k],m[k])\Q[k] , ¥.[e[k]\Q[k\] =¥.[e{c[k],m[k])\Q[ 
Thus, the denominator is common for all terms of inequality (|22]l, and it suffices to prove: 

N 

E [A[k]\Q[k]] <B + Y^ [Xn{b{c[k],m[k]) + I[k]) - ln[k]\Q[k] 

n=l 

To this end, by squaring ( [T8| ) and noting that max[x,0]^ < x^, we have for each n: 

lQn[k + lf < liQnik] + Xn{D[k] + I[k]) - ln[k]f 



(23) 



1 



= ^Qn[k]^ + ^{KiD[k] + I[k]) - ln[k]f + Q.nm\niD[k] + I[k]) - hM 

Summing the above over n G {!,..., N} and using the definition of A[k] gives: 

N N 
AM < - Y.iXn{D[k] + I[k]) - ln[k]f + Qn[k]{Xn{D[k] + I[k]) - ln[k]) 



n=l 



n=l 



Taking conditional expectations given Q[k] and using the bound B proves (23 1. □ 
2 ) The Task Scheduling Algorithm: Our algorithm takes actions every frame to minimize the last two terms on 
the right-hand-side of the drift-plus-penalty ratio bound (22 1. The only part of these terms that we have control 
over on frame k (given the observed Q[k]) is given below: 



E 



Ve{c[k],m [k] ) - En=l Qn [k] ln[k]\Q [k] 



E 



D{c[k],m[k]) + I[k]\Q[k] 



Recall from Lemma [T] that minimizing the above ratio of expectations is accomplished over a deterministic choice 
of (c[A;], m[A:], Thus, every frame k we perform the following: 

^In more detail, by iterated expectations we iiave E [D[fc]|Q[fc]] = E [E [D[k]\{c[k],m[k]), Q[k]] \Q[k]], and E [D[k]\{c[k],m[k]), Q[k]] = 
E [Z)[fc] |(c[fc], m[fc])] because D[k] is conditionally independent of the past given the current (c[fc],m[fc]) used. 
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Observe queues Q[k] = {Qi[k], . . . ,(5jv[A;]). Then choose c[k] G {1, • • • ,N}, m[k] G A4, and I[k] such that 
< I[k] < Imax to minimize: 

Ve{c[k],m[k])-Q,[,][k] 
D{c[k],m[k]) + I[k] 

Update queues Qn[k] for each n e {1, . . . , A^} via ( [TS] ), using the D[k], I[k], and ln[k] values that result from 
the decisions c[k], m[k], I[k] that minimized ([24]) 



3) Steps to minimize ^24^ : Here we elaborate on how to perform the minimization in (|24]) for each frame k. For 
each c € {1, . . . , A^} and m € A4, define idle{c, m) as the value of I[k\ that minimizes (24i, given that we have 
(c[/c], m[/c]) = (c, m). It is easy to see that: 



idle{c, m) 

Now define val{c,m) by: 



ifVe{c,m)-Qc[k]<Q 

Irr^aT. OthcrwisC 



Ve{c,m) - Qc[k] 

val[c,m) — 



D{c, m) + idle{c, m) 

Then we choose (c[A;], m[/i:]) as the minimizer of val{c,m) over c G {1,...,A^} and m G Ai, breaking ties 
arbitrarily, and choose I[k] = idle{c[k],m[k]). Note that this algorithm chooses I[k] = or I[k] = Imax on every 
frame k. Nevertheless, it results in a frame average I that approaches optimality for large V. 



H. Performance of the Task Scheduling Algorithm 

For simplicity, the performance theorem is presented in terms of zero initial conditions. It is assumed throughout 
that the problem (|3]l-([6]l is feasible, so that it is possible to satisfy the constraints. 

Theorem 1: Suppose (5„[0] = for all n G {1, . . . , N}, and that the problem (|3]l-([6]) is feasible. Then under the 
above task scheduling algorithm: 

a) For all frames A' G {1, 2, 3, . . . } we havej^ 

_ ''^^l < power^P^ + - ^ (25) 

D[K] + I[K] - V{D[K] + I[K]) 

where B is defined in Lemma [s] power°P^ is the minimum power solution for the problem Q-Q, and e[K], D[K], 
I[K] are defined by: 



b) The desired constraints ( [20] ) and ( |2T] ) are satisfied for all n G {1, . . . , N}. Further, we have for each frame 
i^G {1,2,3,...}: 

^^-V K ^^^^ 

where ||Q[/c]||4y^^^^^ Qn[kY is the norm of the queue vector (being at least as large as each component Qn[k]), 
and /3 is a constant that satisfies the following for all frames k: 

/3 > E [power°P\D[k\ + I[k]) - e[k]] 

Such a constant /3 exists because the second moments (and hence first moments) are bounded. 

In the special case of a deterministic system, all expectations of the above theorem can be removed, and the 
results hold deterministically for all frames K. Theorem [T] indicates that average power can be pushed arbitrarily 
close to power°P^, using the V parameter that affects an 0{1/V) performance gap given in ( [25] ). The tradeoff is 
that V increases the expected size of E [Qnl^^^]] /K as shown in (|26]l, which bounds the expected deviation from 
the nth constraint during the first K frames (recall ^T9\ from Lemma |2]). Under a mild additional "Slater-type" 
assumption that ensures all constraints can be satisfied with "e-slackness," a stronger result on the virtual queues 

^The right-hand- side in {15\ can be simplified to povjer°^^ + B / iV Dmin), since all frames are at least Dmin in size. 
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can be shown, namely, that the same algorithm yields queues with average size 0{V) [1]. This typically ensures a 
tighter constraint tradeoff than that given in p6l ). A related improved tradeoff is explored in more detail in Section 
lULCl 

Proof: (Theorem [T] part (a)) Given Q[k] for frame k, our control decisions minimize the last two terms in the 
right-hand-side of the drift-plus-penalty ratio bound (|22]), and hence: 



< 



B 



E[A[k] + Ve[k]\Q[k]] 

E[D[k] + I[k]\Q[k]] ^ E[D[k]+I[k]\Q[k]] ' E 



+ 



E[Ve{c*[k],m*[k])\Q[k]] 



D{c*[k],m*[k]) + I*[k]\Q[k] 



+- 



En=lQnm 


'Xn{Dic*[k],m*[k])+r[k])- 


- KMQ[k]_ 


E 


D{c*[k],m*[k]) + I*[k]\Q[k] 





(27) 



where c*[k], m*[k], I*[k], are from any alternative (possibly randomized) decisions that can be made on 

frame k. Now recall the existence of stationary and randomized decisions that yield ([T5])-([T6l). In particular, these 
decisions are independent of queue backlog Q[k] and thus yield (from ([TS])): 

E[eic*[k],m*[k])\Q[k]] _ E[eic*[k],m*[k])] _ _,_opt 



E 



D{c*[k],m*[k]) + I*[k]\Q[k] 



E 



and for all n G {1, ... , N} we have (from ([T6]l): 



E 



E 



XniDic*[k],m*[k]) + r[k]) - ll[k]\Q[k] 

Plugging the above into the right-hand-side of ( |27| ) yields 

E[A[k] + Ve[k]\Q[k 



< 



power 

D{c*[k],m*[k]) + I*[k]' 

Xn{D{c*[k],m*[k]) + r[k])-ll[k] 
+ Vpower°P^ 



< 



B 



E[D[k] + I[k]\Q[k]] - E[D[k]+I[k]\Q[ 
Rearranging terms gives: 

E [A[A:] + Ve[k]\Q[k]] <B + Vpower°P^E [D[k] + I[k\\Q[k]] 

Taking expectations of the above (with respect to the random Q[k]) and using the law of iterated expectations 
gives: 

E [A[fc] + Ve[k]] <B + Vpower°P*E [D[k] + I[k]] (28) 



□ 



The above holds for all k G {0, 1, 2, . . .}. Fixing a positive integer K and summing (28l over k €^ {0,1, . . . ,K — 1} 
yields, by the definition A[k] = L[k + I] - L[k]: 

A'-l K-l 

E [L[K] - L[0]] + V^E [e[k]] < BK + Vpower°^^ E [D[k] + I[k]] 

fe=0 fc=0 

Noting that L[0] = and L[K] > and using the definitions of e[K], DlK], 7[K] yields: 

VKe[K] <BK + VKpower°P\~D[K] + 7[K]) 

Rearranging terms yields the result of part (a). 

Proof: (Theorem [T] part (b)) To prove part (b), note from ( [28] ) we have: 

E [A[k]] < B + VE [power°P\D[k] + I[k]) - e[k]] <B + V/3 

Summing the above over k e {0,1, . . . , K — 1} gives: 

E [L[K]] - E [L[0]] <{B + Vp)K 

Using the definition of L[K] and noting that L[0] = gives: 

K 

J2^[Qn[K]^] < 2{B + V(3)K 
1=1 
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Thus, we have E [HQ [i^] 1 1 2] < 2(S+F/3)iv:. Jensen's inequality for /(x) = ensures E [||Q[A;]||]2 < E [\\Q[ 



and so for all positive integers K: 



E[\\Q[k]\\f<2iB + V(3)K 



(29) 



Taking a square root of both sides of p9| ) and dividing by K proves 
{1,...,JV}: 



From (26 1 we have for each n G 



and hence by Lemma [2]we know constraint (21_) holds. Further, in [15] it is shown that ^29\ together with the fact 
that second moments of queue changes are bounded implies limfc_>oo Qn[k]/k = with probability 1. Thus, ( jlQ] ) 
holds. □ 



/. Simulation 

We first simulate the task sche duling algorithm for the simple deterministic system with one class and one 



constraint, as described in Section II-D The e(m) and D{m) functions are defined in ([8])-(|9]l, and the goal is to 



minimize average power subject to a processing rate constraint 1/{D + I) > 0.2. We already know the optimal 
power is power"^^ = 7/15 0.466667. We expect the algorithm to approach this optimal power as V is increased, 
and to approach the desired behavior of using I[k] = for all k, meeting the constraint with equality, and using 
m[k] = 1 for 1/3 of the frames. This is indeed what happens, although in this simple case the algorithm seems 
insensitive to the V parameter and locks into a desirable periodic schedule even for very low (but positive) V 
values. Using V = 1 and one million frames, the algorithm gets average power 0.466661, uses m[k] = 1 a fraction 
of time 0.333340, has average idle time / = 0.000010, and yields a processing rate 0.199999 (almost exactly equal 
to the desired constraint of 0.2). Increasing V yields similar performance. The constraint is still satisfied when we 
decrease the value of V, but average power degrades (being 0.526316 for V = 0). 
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Fig. 3. Average power versus V. 



Fig. 4. Processing Rate lio /T versus frame index. 



We next consider a system with 10 classes of tasks and two processing modes. The energy and delay characteristics 
for each class i G {1, . . . , 10} and mode m G {1, 2} are: 

Model: (e(i, 1), 1)) = (li, 5i) (30) 
Mode 2: {e{i, 2), D{i, 2)) = (2i, 3i) (31) 

so that mode 1 uses less energy but takes longer than mode 2, and the computational requirements for each class 
increase with i G {1, . . . , 10}. We assume desired rates are given by Aj = p/ (30i) for i G {1, . . . , 10}, for some 
positive value p. The problem is feasible whenever p < I. We use p = 0.8 and run the simulation for 10 million 
frames. Fig. [3] shows the resulting average power as V is varied between and 3, which converges to near optimal 
after V = 0.3. All 10 processing rate constraints are met within 5 decimal points of accuracy after the 10 million 
frames. An illustration of how convergence time is affected by the V parameter is shown in Fig. [4j which illustrates 
the average processing rate lio/T for class 10, as compared to the desired constraint Aio = p/300. It is seen, 
for example, that convergence is faster for V = 0.05 than for y = 1. Convergence times can be improved using 
non-zero initial queue backlog and the theory of place holder backlog in [1], although we omit this topic for brevity. 
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III. Optimization with General Attributes 

This section generalizes the problem to allow time average optimization for abstract attributes. Consider again a 
frame-based system with frame index k G {0, 1,2,.. .}. Every frame k, the controller makes a control action a[k], 
chosen within an abstract set A of allowable actions. The action a[k] affects the frame size T[k] and an attribute 
vector y[k] = {yQ[k],yi[k], . . . ,yL[k]). Specifically, assume these are random functions that are conditionally 
independent of the past given the current a[k] decision, with mean values given by functions T{a) and yi{a) for 
all A. 

f{a)=^[T[k]\a[k]=a] , yi{o^) =¥.[yi[k]\a[k] = a] 

Similar to the previous section, it is assumed there is a minimum frame size Tmin > such that T[k\ > Tmin for 
all k, and that second moments are bounded by a constant o"^, regardless of the policy a[k]. The joint distribution 
of {T[k], yolk], yi[k], . . . ,yL[k]) is otherwise arbitrary. 
Define frame averages T and by: 

^ K-l ^ K-l 



K^oo K ' ' K-^oo K ' 

fc=0 k=0 



As discussed in Section 
problem is then: 



II-B 



the value yjT represents the time average associated with attribute yi[k]. The general 



Minimize: yo/T (32) 

Subject to: yjT <ci V/ G {1, . . . , L} (33) 

a[k] V/c G {0,1,2,...} (34) 

where ci, . . . , are given constants that specify the desired time average constraints. 



A. Mapping to the Task Scheduling Problem 

To illustrate the generality of this framework, this subsection uses the new notation to exactly represent the task 
scheduling problem from Section |ll] For that problem, one can define the control action a[k] to have the form 
a[k] = (c[A;], m[A;], /[A;]), and the action space A is then the set of all (c, m, I) such that c G {1, . . . , A^}, m G M., 
and < / < Imax- 

The frame size is T[k] = D[k] + I[k\, and T{a[k]) is given by: 

f{a[k]) = b{c[k],m[k]) + I[k] 

We then define yo[^] as the energy expended in frame k, so that yo[^] = and yo(«[fc]) = e(c[/c], m[A;]). There 
are N constraints, so define L = A^. To express the desired constraints In/T > An in the form y„/T < Cn, one 
can define yn[k] = —ln[k] and Cn = — A„ for each n G {1,...,A^}, and yn{a[k]) = —ln[k]. Alternatively, one 
could define yn[k] = ^nT[k] — ln[k] and enforce the constraint < for all n G {1, . . . , A'^}. 

This general setup provides more flexibility. For example, suppose the idle state does not use energy, but 
operates at a low power pidie and expends total energy piiiiel[k] on frame k. Then total energy for frame k can be 
defined as yo[k] = e[k] + pidieli^^ where e[k] is the energy spent in the busy period. The setup can also handle 
systems with multiple idle mode options, each providing a different energy savings but incuning a different wakeup 
time. 



B. The General Algorithm 



The algorithm for solving the general problem (|32|)-p4|) is described below. Each constraint yi < ciT in ( [33) is 
treated using a virtual queue Qi [k] with update: 



Qi[k + 1] = max[Q,[A;] + yi[k] - ciT[k],0] V/ G {1, . . . ,L} 



(35) 
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Defining L{k] and A[k] as before (in Section II-G[ ) leads to the following bound on the drift-plus-penalty ratio, 
which can be proven in a manner similar to Lemma [3j 



K[A[k] + Vyo[k]\Q[k]] 



< 



B 



+ 



E 


VyoHk]) + Zti Qi[k]m 


{a[k])\Q[k] 


E 


fia[k])\Q[k] 





E[T[k]\Q[k]] - E[T[k]\Q[k]] 

where i? is a constant that satisfies the following for all Q[k] and all possible actions a[k]: 

1 ^ 

B>^Y.^[{m[k]-ciT[k]f\Q[k]] 



(36) 



1=1 



Every frame, the controller observes queues Q[k] and takes an action a[k] G ^ that minimizes the second term 
on the right-hand-side of (36 1. We know from Lemma [T] that minimizing the ratio of expectations is accomplished 
by a deterministic selection of a[k] G A. The resulting algorithm is: 

• Observe Q[k] and choose a[k] ^ A to minimize (breaking ties arbitrarily): 



vyoic^m + Ei=iQi[mHk]) 



T{a[k]) 

Update virtual queues Qi[k] for each I € {1, . . . , L} via (|35 



(37) 



One subtlety is that the expression p7] ) may not have an achievable minimum over the general (possibly infinite) 
set A (for example, the infimum of the function f{x) = x over the open interval < x < 1 is not achievable over 
that interval). This is no problem: Our algorithm in fact works for any approximate minimum that is an additive 
constant C away from the exact infimum every frame k (for any arbitrarily large constant C > 0). This effectively 
changes the "B" constant in our performance bounds to a new constant "B + C" [1]. Let ratio°P^ represent the 



optimal ratio o{ Ijq/T for the problem (|32])-(34i. As before, it can be shown that if the problem is feasible (so that 
there exists an algorithm that can achieve the constraints ([33])-([34l)), then any C-additive approximation of the above 
algorithm satisfies all desired constraints and yields y^/T < ratio'^^ + 0{l/V), which can be pushed arbitrarily 
close to ratio"^*^ as V is increased, with the same tradeoff in the queue sizes (and hence convergence times) with 
V. The proof of this is similar to that of Theorem [T] and is omitted for brevity (see [1][2] for the full proof). 



C. Random Task Arrivals and Flow Control 



Admit 



Y2[k] 



AJk] 



AJk] 



D 






Processor 









AJk] 



l\|Y Completed tasks 



Drop 

Fig. 5. A task processing system with random arrivals and flow control. 



Again consider a system with classes of tasks, as in Section |lT] Each frame k again has a busy period of 
duration D[k] and an idle period of duration I[k] as in Fig. [T] However, rather than always having tasks available 
for processing, this subsection assumes tasks arrive randomly with rates (Ai, . . . , Xn), where A„ is the rate of task 
arrivals per unit time (see Fig. [Sjl. At the beginning of each busy period, the controller chooses a variable c[k] 
that specifies which type of task is performed. However, c[k] can now take values in the set {0, 1, ... , N}, where 
c[k] = is a null choice that selects no task on frame k. If c[k] = 0, the busy period has some positive size Dq and 
may spend a small amount of energy to power the electronics, but does not process any task. The mode selection 
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variable m[k] takes values in the same set A4 as before. The idle time variable I[k] is again chosen in the interval 

[0; Imax] ■ 

Further, for each n G {1, 2, . . . , N} we introduce ^ow control variables "fn[k], chosen in the interval < jnik] < 
1. The variable 7n[A;] represents the probability of admitting each new randomly arriving task of class n on frame 
k (see Fig. |5]). This enables the system to drop tasks if the raw arrival rates (Ai, . . . , Aat) cannot be supported. Let 
'y[k] = (7i[A;], . . . ,7Ar[/c]) be the vector of these variables. 

We thus have a[k] = {c[k],m[k], I[k],'y[k]), with action space A being the set of all {c,m,I,'y) such that 
c G {0, 1, . . . , N}, m e M, < I < Imax, and < 7„ < 1 for n G {1, . . . , N]. Define e[k] and D[k] as the 
energy and busy period duration for frame k. Assume e[k] depends only on (c[A:], m[A;], /[A;]), and D[k] depends 
only on (c[A;], m[/i;]), with averages given by functions e{c[k],m[k], I[k]) and D{c[k],m[k]): 

e{c[k],m[k]J[k]) =¥.[e[k]\c[k],m[k],I[k\\ , b{c[klm[k]) =¥.[D[k]\c[k],m[k]] 

Finally, for each n G {!,..., A^}, define An[k] as the random number of new arrivals admitted on frame k, which 
depends on the total frame size D[k] +I[k] and the admission probability 7n[A;]. Formally, assume the arrival vector 
. . . ,AN[k\) is conditionally independent of the past given the current a[k] used, with expectations: 

^[An[k]\a[k]] = Xr,^n[k][b{c[klm[k]) + I[k]] (38) 

The assumption on independence of the past holds whenever arrivals are independent and Poisson, or when all 
frame sizes are an integer number of fixed size slots, and arrivals are independent and identically distributed (i.i.d.) 
over slots with some general distribution. 

We seek to maximize a weighted sum of admission rates subject to supporting all of the admitted tasks, and to 
maintaining average power to within a given positive constant Pav'- 

Maximize: "^^^ (39) 

D + I 

Subject to: ]4„/(;D + 7) < T„/(I) + 7) Vn G {1, . . . , A^} (40) 

e 



< Pav (41) 

D + I- 

a[k] G ^ VA: G {0,1,2,...} (42) 

where {wi, . . . ,wn) are a collection of positive weights that prioritize the different classes in the optimization 
objective. 

We thus have L = + 1 constraints. To treat this problem, define T[k\ = D[k\ + I[k], so that T{a[k]) = 
D{c[k\,m[k]) + I[k]. Further define yo[^]> yn[k\ for n G {1, . . . , N}, and x[k\ as: 

TV N 

yo[k] = -^tfn^n[A;] =^ yo{a[k]) = - WnXnlnjk] [D{c[k],m[k]) + I[k]] 

n=l n=l 

yn[k] = An[k] - ln[k] =^ y„ (o [fc] ) = A„7n [fc] [D(c[fc] , m[A:] ) + /[A:]] - 1„ [A:] 
x[k] = e[k] - [D[k]+I[k]]Pav =^ x{a[k]) = e{c[klm[k], I[k]) - [b{c[k],m[k]) + I[k]]Pav 

Then: 

• Minimizing y^/T is equivalent to (|39]l. 

• The constraints y„ < for n G {1, . . . , N} are equivalent to ( |40l ). 

• The constraint x < is equivalent to ( |4T| ). 

Note that the above problem does not specify any explicit queueing for the randomly arriving tasks. The algorithm 
will in fact construct explicit queues (so that the virtual queues can be viewed as actual queues). Note also that 
the constraint a [A;] ^ A does not allow restrictions on actions based on the queue state, such as when the queue 
is empty or not. Thus, in principle, we allow the possibility of "processing" a task of class n even when there 
is no such task available. In this case, we assume this processing is still costly, in that it incurs time equal to 
L'(c[A;], m[A;]) and energy equal to e(c[A;], m[A;], /[A;]). Our algorithm will naturally learn to avoid the inefficiencies 
associated with such actions. 
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1) The Dynamic Algorithm for Random Task Arrivals: To enforce the constraints y„ < for each n G 
{!,..., N}, define queue Qn[k\ with update: 

Qn[k + 1] = max[Q„[fc] + An[k] - 0] (43) 

To enforce x < 0, define a virtual queue Z[k] with update: 

Z[k + 1] = max[Z[A;] + e[k\ - [D[k] + I[k]]Pa^, 0] (44) 



It can be seen that the queue update ( |43) is the same as that of an actual queue for class n tasks, with random 
task arrivals An{k] and task service The minimization of ( [37] ) then becomes the following: Every frame k, 

observe queues Q[k] and Z[k]. Then choose c[k] G {0, 1, ... , N}, m[k] G A^, I[k] G [0, /max]> and 7n[A;] G [0, 1] 
for all n G {1, ... , N} to minimize: 

En=l Wn\nln[k][b{c[k],m[k]) + /[fc]] 



L»(c[A:],m[A;]) + 



I En=l QnmXnln[k][D{c[k]Mk\) + ^W] - In W) 

b{c[k],m[k]) + I[k] 
^ Z[fc](e(c[fc],m[fc],/[A;])-[^(c[fc],m[fc]) + I[fc]]Pa.) 
D(c[A;],m[A:]) + /[A;] 

After a simplifying cancellation of terms, it is easy to see that the 7„[fc] decisions can be separated from all 
other decisions (see Exercise |2]). The resulting algorithm then observes queues Q[k] and Z[k\ every frame k and 
performs the following: 

• (Flow Control) For each n G {1, . . . , N}, choose 7n[fc] as: 



(Task Scheduling) Choose c[k] G {0, 1, . . . , A^}, m[k] G M, I[k] G [0, Imax] to minimize: 



Z[k]e{c[k]Mk],m)-T.n=lQnmn[k] 



(47) 



D{c[k],m[k]) + I[k] 

• (Queue Update) Update Qn[k] for each n G {1, . . . , iV} by (|43), and update Z[k] by (|44]|. 
The minimization problem ( [47] ) is similar to ( |24| ), and can be carried out in the same manner as discussed in 
Section II-G| A key observation about the above algorithm is that it does not require knowledge of the arrival rates 



(Ai, . . . , Aat). Indeed, the A„ terms cancel out of the minimization, so that the flow control variables 7n[^] in (46 1 
make "bang-bang" decisions that admit all newly arriving tasks of class n on frame k if Qn[k] < Vwn, and admit 
none otherwise. This property makes the algorithm naturally adaptive to situations when the arrival rates change, 
as shown in the simulations of Section IIII-DI 

Note that if e(0, m, /) < e(c, m, /) for all c G {1, . . . , N}, m G M, / G [0 

1 Imax]t SO that the energy associated 



with processing no task is less than the energy of processing any class c / 0, then the minimization in ( |47) will 
never select a class c such that Qc[k] = 0. That is, the algorithm naturally will never select a class for which no 
task is available. 

2 ) Deterministic Queue Bounds and Constraint Violation Bounds: In addition to satisfying the desired constraints 
and achieving a weighted sum of admitted rates that is within 0{1/V) of optimality, the flow control structure of 
the task scheduling algorithm admits deterministic queue bounds. Specifically, assume all frame sizes are bounded 
by a constant Tmax, and that the raw number of class n arrivals per frame (before admission control) is at most 
An,max- By the flow control policy (|46]), new arrivals of class n are only admitted if Qn[k] < Vwn- Thus, assuming 
that Qn[0] < Vwn + An^max, wc must havc Qn[k] < ywn + An-max for all frames A;G{0,1,2,...}. This specifies a 
worst-case queue backlog that is 0{V), which establishes an explicit [0{1/V), 0{V)\ performance-backlog tradeoff 
that is superior to that given in (|26]l. 

With mild additional structure on the e(c, m,/) function, the deterministic bounds on queues Qn[k] lead to a 
deterministic bound on Z[k], so that one can compute a value Zmax, of size 0{V), such that Z[k] < Zmax for all 
k. This is explored in Exercise [3] (see also [14]). 
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D. Simulations and Adaptiveness of Random Task Scheduling 



Here we simulate the dynamic task scheduling and flow control algorithm (46l-(47i, using the 10-class system 



model defined in Section II-I with e(c, m) functions given in (|30l)-(|3T]). For consistency with that model, we remove 
the c[k] =0 option, so that the decision ( |3T| ) chooses c[k\ = l,m[k] = 1, incurring one unit of energy, in case 
no tasks are available. Arrivals are from independent Bernoulli processes with rates Aj = /9/(30i) for each class 
i S {!,..., 10}, with p = 0.8. We use weights Wn = ^ for all n, so that the objective is to maximize total 
throughput, and Pav = 0.5, which we know is feasible from results in Fig. [3] of Section II-I Thus, we expect 
the algorithm to learn to admit everything, so that the admission rate approaches the total arrival rate as V is 
increased. We simulate for 10 million frames, using V in the interval from to 200. Results are shown in Figs. 
|6] and |7] Fig [6] shows the algorithm learns to admit everything for large V (100 or above), and Fig. |7] plots the 
resulting average queue size (in number of tasks) per queue, together with the deterministic bound Q„ [k] <V + 60 
(where An,max = 60 in this case because there is at most one arrival per slot, and the largest possible frame is 
Dmax + Imax = 50 + 10 = 60). The average power constraint was satisfied (with slackness) for all cases. The 
average queue size in Fig. [7] grows linearly with V until V = 100, when it saturates by admitting everything. The 
saturation value is the average queue size associated with admitting the raw arrival rates directly. 
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Fig. 6. Total admission rate versus V. 



Fig. 7. Average queue size per queue, and tiie 
deterministic bound from Section Illl-C.21 



We now illustrate that the algorithm is robust to abrupt rate changes. We consider the same system as before, 
run over 10 million frames with V = 100. However, we break the simulation timeline into three equal size phases. 
During the first and third phase, we use arrival rates Aj = pi/{30i) for i G {1, . . . , 10}, where pi = 0.8 as in 
the previous experiment. Then, during the second (middle) phase, we double the rates to Aj = p2/{30i), where 
P2 = 1.6. Because p2 > 1, these rates are infeasible and the algorithm must learn to optimally drop tasks so as to 
maximize the admission rate subject to the power constraint. Recall that the algorithm is unaware of the anival rates 
and must adapt to the existing system conditions. The results are shown in Figs. [8] and |9] Fig. [8] shows a moving 
average admission rate versus time. During the first and third phases of the simulation, we have pi = 0.8 and 
the admitted rates are close to the raw arrival rate (shown as the lower dashed horizontal line). During the middle 
interval (with p2 = 1.6), the algorithm quickly adapts to the increased arrivals and yields admitted rates that are 
close to those that should be used in a system with loading p2 = 1.6 always (shown as the higher dashed horizontal 
line). Fig. [9] plots the corresponding moving average queue backlog per queue. The lower dashed horizontal line 
indicates the value of average backlog that would be achieved in a system with loading pi = 0.8 always. Also 
shown is the deterministic queue bound y + 60 = 160, which holds for all frames, regardless of the raw arrival 
rates. 



E. Task Scheduling: Extensions and Further Reading 

This section considered minimizing time averages subject to time average constraints. Extended techniques for 
minimizing convex functions of time average vectors (or maximizing concave functions) subject to similar constraints 
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are treated in the renewal optimization theory of [1][2]. This is useful for extending the flow control problem (39 1 



(42 1 to optimize a sum of concave utility functions (j)n{x) of the time average admission rates for each class: 

N 

Y,4>n {An/(D + I)) 
n=l 

Using logarithmic functions (t)n{x) leads to proportional fairness [16], while other fairness properties are achieved 
with other concave functions [17] [18] [19]. Related utility optimization for particular classes of wireless systems 
with fixed size frames of T slots, and with probabilistic reception every slot, is treated using a different technique 
in [20][21]. Related work on utility optimal scheduling for single-slot data networks is treated using convex 
optimization in [22] [23] [24] [25], stochastic "primal-dual" algorithms in [26] [27] [28] [29] [30], and stochastic "dual" 
algorithms in [12][31][32][13][33]. 

For the problem with random arrivals, queueing delay can often be improved by changing the constraint < In 



of ( [40| ) to An + e < 1„, for some e > 0. This specifies that the output processing rate should be e larger than the 
arrival rate. However, unlike the case e = 0, such a constraint yields a queue Qn[k] that contains some "fake" tasks, 
and can lead to decisions that serve the queue when no actual tasks are available. This can be overcome using the 
theory of e-persistent service queues in [34] [35]. 

Note that our flow control algorithm rewards admission of new data on each frame. This fits the general framework 
of this section, where attributes yi[k] are (possibly random) functions of the control action a[k\ G A. One might 
attempt to solve the problem by defining a reward upon completion of service. This does not fit our framework: 
That is because the algorithm might "complete" a service in a queue that is empty, simply to accumulate the "fake" 
reward. One could eliminate fake rewards by an augmented model that allows rewards yi{a[k], Q[k]) and/or action 
spaces A{Q[k]) that depend on the current backlog, although this creates a much more complex Markov decision 
model that is easily avoided by defining rewards at admission. However, there are some problems that require 
such a reward structure, such as problems of stock market trading where prior ownership of a particular stock is 
required to reap the benefits of selling at an observed desirable price. These problems can be treated with a modified 
Lyapunov function of the form L[k] = X]^=i(Qn[^] ~ ^)^' which pushes backlog towards a non-empty state 6. 
This approach is used for stock trading [36], inventory control [37], energy harvesting [38], and smart-grid energy 
management [39]. The first algorithms for optimal energy harvesting used a related technique [40], and related 
problems in processing networks that assemble components by combining different sub-components are treated in 
[41] [42]. 



F. Exercises for Section III 



Exercise 1: Consider a system with classes, each of which always has new tasks available. Every frame 
k we choose a class c[k] € {1,...,A^}, and select a single task of that class for processing. We can process 
using mode m[k] G A4. The result yields frame size T{c[k],m[k]), energy e{c[k],m[k]), and processing quality 
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q{c[k],m[k]). Design an algorithm that selects c[k] and m[k] every frame k to maximize q/T subject to an average 
power constraint e/T < Pav and to a processing rate constraint of at least A„ for each class n. 

Exercise 2: Show that minimization of (|45|) leads to the separable flow control and task scheduling algorithm of 



Exercise 3: Consider the flow control and task scheduling algorithm (|46l)-(47 1, and recall that Qn[k] < Qmax 
for all 77. S {1, ... , A^} and all frames k, where Qmax= ™ax^gM [Vwn + An^max]- Suppose there is a constant 



ne{l,...,N}[ 

emin > such that e(c, m, /) > Cmin for all c / 0, and that e(0, m, /) = for all m G M, I G [0, /, 



) ^max\ ■ 



a) Show that the minimization of ( [47] ) chooses c[k] = whenever Z[k] > Qmax/^min- 

b) Suppose there is a constant Cmax such that e[k] < Cmax for all k. Conclude that Z[k] < Zmax=Qmaxl ^min + 
e-max for all frames k. 

c) Use the queueing equation (44i to show that over any sequence of K frames {j, j + 1, • . . , j + — 1} (starting 
at some frame j), the total energy usage satisfies: 

where T[k] is the size of frame k. Hint: Make an argument similar to the proof of Lemma |2] 

Exercise 4: Consider the same general attributes yi[k] of Section [51} with yi{a[k]) for a[k] G A. State the 
algorithm for solving the problem of minimizing yg subject to yi/T < ci. Hint: Define appropriate attributes 
xi[k] on a system with an "effective" frame size of 1 every frame, and minimize xq/I subject to < for 
/G{1,...,L}. 

Exercise 5: Modify the task scheduling algorithm of Section II-G to allow the controller to serve more than one 
task per frame. Specifically, every frame k the controller chooses a service action s[k] from a set S of possible 
actions. Each service action s G 5 determines a clearance vector c{s) = (ci(s), . . . , CAr(s)), where Cj(s) is the 
number of tasks of type i served if action s is used on a given frame. It also incurs a delay D{s) and energy e(s). 

Exercise 6: Consider the linear fractional problem of finding a vector (xi, . . . ,xm) to solve: 

Minimize: °»+^^^"-"- 



Subject to: YlfLi di^t < di V/ G {1, . . . , L} 
0<Xi<l V« G {1,...,M} 

Assume constants a^, bi, cu, di are given, that 6o > 0, and that 6j > for all i G {1, . . • Treat this as a 

time average problem with action a[k] = (xi[/i:], . . . , XM[k]), action space A = {(xi, . . . , xj\/)|0 < Xj < 1 Vi G 
{!,..., M}}, frame size T[k] = 6o + Y^'iLi^iXilk], and where we seek to minimize (ao + YldLi^^iXi) / {ho + 
Y.'^i biXi) subject to Y^iii Cii^i < di for all / G {1, ... , L}. 

a) State the drift-plus-penalty ratio algorithm ( [37] ) for this, and conclude that: 

for some finite constant B, where ratio"^^ is the optimal value of the objective function, and Xi[K]=^ J2k=o 
Thus, the limiting time average satisfies the constraints and is within B/V from optimality. Your answer should 
solve for values (j)i[k] such that on frame k we choose . . . ,XM[k]) over A to minimize "^"W+^gi . 

Note: In Appendix B it is shown that this minimization is performed as follows: Define I={i G {1, . . . , M}\bi = 0} 
and J^{j G {1, . . . , M}\bj > 0}. For all i G I, choose Xi[k] = if cpiik] > 0, and Xi[k] = 1 if (j)i[k] < 0. Next, 
temporarily select Xj[k] = for all j G J^. Then rank order the indices j € J' from smallest to largest value of 
4>j[k]/bj, and, using this order, greedily change Xj[k] from to 1 if it improves the solution. We stop when we 
reach the first index in the rank ordering that does not improve the solution. 

b) Note that the case bo = 1 and 6j = for f 7^ is a linear program. Give an explicit decision rule for each 
Xi[k] in this case (the solution should be separable for each i G {1, . . . , M}). 
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IV. Reacting to Randomly Observed Events 

Consider a problem with general attributes yo[^])yi[fc]) ■ ■ ■ iVblk] ^^^^ frame size T[k] for each frame k, as in 
Section III However, now assume the controller observes a random event uj[k\ at the beginning of each frame k, 
and this can influence attributes and frame sizes. The value of oj[k] can represent a vector of channel states and/or 
prices observed for frame k. Assume {tij[A;]}^Q is independent and identically distributed (i.i.d.) over frames. The 
controller chooses an action a[k] € where the action space ^(cij[A;]) possibly depends on the observed 

oj[k\. Values of (T[/c], yol^li ■ ■ ■ TUhik]) are conditionally independent of the past given the current ijj[k] and a[k], 
with mean values: 

U^[k],a[k])=¥.[yi[k]\u[k],a[k]] , f{u[kla[k])=¥.[T[k]\uj[kla[k]] 

The goal is to solve the following optimization problem: 

Minimize: y^/T (48) 
Subject to: yjT < ci V/ G {1, . . . , L} (49) 

a[k] £ A{oj[k]) yk £ {0,1,2,...} (50) 

As before, the constraints yi < ciT are satisfied via virtual queues <5;[A;] for / G {1, . . . , L}: 

Qi[k + 1] = ma^Qiik] + yi[k] - ciT[k],0] (51) 

The random uj[k] observations make this problem more complex that those considered in previous sections of this 
paper. We present two different algorithms from [1][2]. 

Algorithm 1: Every frame k, observe Q[k] and uj[k] and choose a[k] G ^(^[A;]) to minimize the following ratio 
of expectations: 



E 


VyoHk],a[k]) + Zti Qiimi^ 


[k],a[k])\Q[k] 


E 


fico[k],a[k])\Q[k] 





(52) 



Then update the virtual queues via pT) . 

Algorithm 2: Define 9[0] = 0, and define 9[k] for k G {1,2,3, . . .} as a running ratio of averages over past 
frames: 

oimEtoyoWZtom (53) 

Then every frame k, observe Q[k], 6[k], and u}[k], and choose a[k] G to minimize the following function: 



1 



(54) 



Then update the virtual queues via (51 1. 



A. Comparison of Algorithms 1 and 2 

Both algorithms are introduced and analyzed in [1][2], where they are shown to satisfy the desired constraints 
and yield an optimality gap of 0{\/V). Algorithm 1 can be analyzed in a manner similar to the proof of Theorem 
[T] and has the same tradeoff with V as given in that theorem. However, the ratio of expectations ( [52] ) is not 
necessarily minimized by observing uj[k\ and choosing a[k\ G ^(a;[/!;]) to minimize the deterministic ratio given 
(jj[k]. In fact, the minimizing policy depends on the (typically unknown) probability distribution for u}[k\. A more 
complex bisection algorithm is needed for implementation, as specified in [1][2]. 

Algorithm 2 is much simpler and involves a greedy selection of a[k\ G ^((^[fc]) based only on observation of 
a; [k] , without requiring knowledge of the probability distribution for uj [k] . However, its mathematical analysis does 
not yield as explicit information regarding convergence time as does Algorithm 1. Further, it requires a running 
average to be kept starting at frame 0, and hence may not be as adaptive when system statistics change. A more 
adaptive approximation of Algorithm 2 would define the average 9[k] over a moving window of some fixed number 
of frames, or would use an exponentially decaying average. 
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Both algorithms reduce to the following simplified drift-plus-penalty rule in the special case when the frame 
size T(a;[A:], a[/c]) is a fixed constant T for all (^[A;], a[A;]: Every frame k, observe u:[k] and Qlk] and choose 
a[k] e to minimize: 

Vyo {io [k] ,a[k]) + j:t^Qi [k]yi {u [k] ,a[k]) (55) 

Then update the virtual queues via ( |5T] ). This special case algorithm was developed in [14] to treat systems with 
fixed size time slots. 

A simulation comparison of the algorithms is given in [2]. The next subsection describes an application to energy- 
aware computation and transmission in a wireless smart phone. Exercise |7] considers opportunistic scheduling where 
wireless transmissions can be deferred by waiting for more desirable channels. Exercise [8] considers an example of 
price-aware energy consumption for a network server that can process computational tasks or outsource them to 
another server. 

B. Efficient Computation and Transmission for a Wireless Smart Device 

Consider a wireless smart device (such as a smart phone or sensor) that always has tasks to process. Each task 
involves a computation operation, followed by a transmission operation over a wireless channel. On each frame k, 
the device takes a new task and looks at its meta-data fi[k\, being information that characterizes the task in terms 
of its computational and transmission requirements. Let d represent the time required to observe this meta-data. 
The device then chooses a computational processing mode m[k] € M.{fi[k]), where M.[j5[k]) is the set of all mode 
options under I3[k]. The mode m[k] and meta-data fi[k] affect a computation time Dcomp[k], computation energy 
Ccompik], computation quality q[k], and generate A[k] bits for transmission over the channel. The expectations of 
Dcomp[k], &comp\k\, (l\k\ are: 

bcomp{P[k],m[k]) = ¥.[Dcomp[k]\l3[klrn[k]] 
e-comp {fi[k],m[k]) = E [ecomp [k] \l3[k],m[k]] 
q{P[k],m[k]) = ¥.[q[k]\^[k],m[k]] 

For example, in a wireless sensor, the mode m[k\ may represent a particular sensing task, where different tasks can 
have different qualities and thus incur different energies, times, and A[k] bits for transmission. The full conditional 
distribution of A[k\, given fi[k], m[k], will play a role in the transmission stage (rather than just its conditional 
expectation). 

The A[k] units of data must be transmitted over a wireless channel. Let S[k] be the state of the channel on frame 
k, and assume S[k] is constant for the duration of the frame. We choose a transmission mode g[k] G Q, yielding a 
transmission time Dtran[k] and transmission energy etran[k] with expectations that depend on S[k], g[k], and A[k]. 
Define random event uj[k] = {/3[k],S[k]) and action a[k] = {m[k], g[k]). We can then define expectation functions 

Dtrani(^[k],a[k]) and Hk],a[k]) by: 

Dtran iu}[k],a[k]) = E [Dtran [k]\uj[k] , a[k]] , etran {uj[k],a[k]) = E [etran [k] \uj[k],a[k]] 

where the above expectations are defined via the conditional distribution associated with the number of bits A[k] 
at the computation output, given the meta-data /3[fe] and computation mode m[k] selected in the computation 
phase (where P[k] and m[k] are included in the Lo[k], a[k] information). The total frame size is thus T[k] = 
d + ^comp 

[k] + Dtran[k]. 

The goal is to maximize frame processing quality per unit time q/T subject to a processing rate constraint 
of l/T > A, and subject to an average power constraint {ccomp + etran) /T < Pav (where A and Pav are given 
constants). This fits the general framework with observed random events uj[k] = (/3[A:], ^[A;]), control actions 
a[k] = {m[k],g[k]), and action space ^(cli[A;]) = M{fi[k\) x Q. We can define = ViW\ = T[k] - 1/A, 

and y2[k] = ecomp[k] + etran[k] — PavT[k], and solve the problem of minimizing yQ/T subject to < and 
2/2 ^ 0. To do so, let Q[k\ and Z[k] be virtual queues for the two constraints: 

Q[k + l] = max[Q[A;] +T[A;] - 1/A,0] (56) 

Z[k + l] = UiB.^[Z[k]+ecamp[k]+etran[k]-PavT[k],^] (57) 
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Using Algorithm 2, we define 6[0] = and 9[k] for k G {1,2, . . .} by ([53]). The Algorithm 2 minimization ([54]) 
amounts to observing {P[k], S[k]), Q[k], Z[k], and 9[k] on each frame k and choosing G A^(/3[fc]) and 

5 [A;] G ^ to minimize: 

V[-q{m,m[k])-0[k]{d + D^ompW [k],m[k]) + Aran Hk],a[k]))] 
+Q[k][d + D,ompim,m[k])+DtranHk],a[k]) - 1/A] 

+Z[k][e 

comp 

{f3[k],m[k]) + [k],a[k])- Pa^{d + DcompW [k],m[k]) + Aran Hk],a[k]))] 

The computation and transmission operations are coupled and cannot be separated. This yields the following 
algorithm: Every frame k: 

• Observe uj[k] = {P[k],S[k]) and values Q[k], Z[k], 9[k]. Then jointly choose action m[k] G A4{(3[k]) and 
g[k] G Q, for a combined action a[k] = {m[k], g[k]), to minimize: 

-Vq{/3[k].m[k]) + D^mp{m,Hk])[Q[k] - Ve[k] - PavZ[k]] + Z[k]ecomp{m,m[k]) 

+DtranHk],a[k])[Q[k] - Ve[k] - PavZ[k]] + Z[k]etranHk], a[k]) 

. (Updates) Update Q[k], Z[k], e[k] via ([56]), (|57|), and ([53). 

Exercise [9] shows the algorithm can be implemented without 9[k] if the goal is changed to maximize q, rather 
than q/T. Further, the computation and transmission decisions can be separated if the system is modified so that 
the bits generated from computation are handed to a separate transmission layer for eventual transmission over the 
channel, rather than requiring transmission on the same frame, similar to the structure used in Exercise [8] 



C. Exercises for Section IV 



Exercise 7: (Energy-Efficient Opportunistic Scheduling [14]) Consider a wireless device that operates over fixed 
size time slots k G {0, 1,2, . . .}. Every slot k, new data of size A[k] bits arrives and is added to a queue. The 
data must eventually be transmitted over a time-varying channel. At the beginning of every slot k, a controller 
observes the channel state S[k] and allocates power p[k] for transmission, enabling transmission of p[k] bits, 
where ii[k] = ji{S[k],p[k]) for some given function fi{S,p). Assume p[k] is chosen so that < p[k] < Pmax for 
some constant Pmax > 0. We want to minimize average power p subject to supporting all data, so that A < Jl. 
Treat this as a problem with all frames equal to 1 slot, observed random events u}[k] = {A[k], S[k]), and actions 
a[k] = p[k] G [0,Pmax]- Design an appropriate queue update and power allocation algorithm, using the policy 
structure 
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Fig. 10. The client/server system of Exercise [sj witli decision variables (7[fc],e[fc]) and observed prices ((/>[fc], 



Exercise 8: (Energy Prices and Network Outsourcing) Consider a computer server that operates over fixed length 
time slots k G {0, 1,2,.. .}. Every slot k, a new task arrives and has size S[k] (if no task arrives then S[k] = 0). 
The server decides to either accept the task, or outsource it to another server (see Fig. 10 1. Let 7 [A;] be a binary 
decision variable that is 1 if the server accepts the task on slot k, and zero else. Define A[k] = 7[A;]5[A;] as the 
total workload admitted on slot k, which is added to the queue of work to be done. Let (j)[k] be the (possibly 
time- varying) cost per unit size for outsourcing, so the outsourcing cost is Cout[k] = '?^[A;](1 — 7[/c])5[A;]. Every 
slot k, the server additionally decides to process some of its backlog by purchasing an amount of energy e[k] at 
a per-unit energy price tplk], serving fi[k] = jl{e[k]) units of backlog with cost Cenergy[k] = ip[k]e[k], where fi{e) 
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is some given function. Assume e[k] is chosen in some interval < e[A;] < Cmax- The goal is to minimize time 
average cost Cout + Cinergy subject to supporting all tasks, so that A <Jl. Treat this as a problem with all frames 
equal to 1 slot, observed random events uj[k] = {S[k], (l}[k],tlj[k]), actions a[k] = {'y[k],e[k]), and action space 
^(a;[A;]) being the set of all (7, e) such that 7 G {0, 1} and < e < Cmax- Design an appropriate queue and state 
the dynamic algorithm ( [SS] ) for this problem. Show that it can be implemented without knowledge of the probability 
distribution for (^[A;], V'i^]), and that the 7[fc] and e[k] decisions are separable. 

Exercise 9: Consider the same problem of Section [TV-B[ with the modified objective of maximizing q subject to 
1/T > A and {ecomp + &tran)/T < Pav Use the observation from Exercise H that this can be viewed as a problem 



with an "effective" fixed frame size equal to 1, and give an algorithm from the policy structure (55 1. 



V. Conclusions 

This paper presents a methodology for optimizing time averages in systems with variable length frames. Ap- 
plications include energy and quality aware task scheduling in smart phones, cost effective energy management 
at computer servers, and more. The resulting algorithms are dynamic and often do not require knowledge of the 
probabilities that affect system events. While the performance theorem of this paper was stated under simple i.i.d. 
assumptions, the same algorithms are often provably robust to non-i.i.d. situations, including situations where the 



events are non-ergodic [1]. Simulations in Section III-D show examples of how such algorithms adapt to changes 



in the event probabilities. Exercises in this paper were included to help readers learn to design dynamic algorithms 
for their own optimization problems. 

The solution technique of this paper uses the theory of optimization for renewal systems from [1][2], which applies 
to general problems. Performance for individual problems can often be improved using enhanced techniques, such 
as using place-holder backlog, exponential Lyapunov functions, LIFO scheduling, and e-persistent service queues, 
as discussed in [1] and references therein. However, the drift-plus-penalty methodology described in this paper 
provides much of the insight needed for these more advanced techniques. It is also simple to work with and 
typically yields desirable solution quality and convergence time. 

Appendix A — Bounded Moment Convergence Theorem 

This appendix provides a bounded moment convergence theorem that is often more convenient than the standard 
Lebesgue dominated convergence theorem (see, for example, [43] for the standard Lebesgue dominated convergence 
theorem). We are unaware of a statement and proof in the literature, so we give one here for completeness. Let 
X{t) be a random process defined either over non-negative real numbers i > 0, or discrete time t G {0, 1,2,.. .}. 
Recall that X{t) converges in probability to a constant x if for all e > we have: 

lim Pr[\X{t) - x| > e] = 

Theorem 2: Suppose there is a real number x such that X{t) converges to x in probability. Further suppose there 
are finite constants C > 0, 5 > such that for all t we have E < C. Then linit^oo E [X{t)] = x. 

Proof: Without loss of generality, assume x = (else, we can define Y{t) = X{t) — x). Fix e > 0. By definition 
of X{t) converging to in probability, we have: 

lim Pr[|X(t)| > e] = (58) 

Further, for all t we have E [X{t)] < E and so: 

K [X{t)] < e + E [\X{t)\ I \X{t)\ > e] Pr[\X{t)\ > e] (59) 

We want to show the final term in the right-hand-side above converges to when f — )• 00. To this end, note that 
for all t we have: 

C > E [|X(t)|^+'^ 

> E [|X(t)|i+^ I \X{t)\ > el Pr[\X{t)\ > e] 

> E[\X{t)\ I \X{t)\ > e]'+'Pr[\X{t)\ > e] (60) 
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where ( [6OI 1 follows by Jensen's inequality applied to the conditional expectation of the function f{\X{t)\), where 
f{x) = x^~^^ is convex over x > 0. Multiplying inequality ( |60| ) by Pr[|X(t)| > e]'' yields: 

CPr[\X{t)\ > e]' > {E[\Xit)\ \ \X{t)\ > e]Pr[|X(t)| > e])'+' > 

Taking a limit of the above as t — )• 00 and using dSSll yields: 



> lim (E [\X{t)\ I \X{t)\ > e] Pr[\X{t)\ > e]^^^ > 

t—^oo 

It follows that: 



lim E [\X{t)\ I \X{t)\ > e] Pr[\X{t)\ > e] = 



Using this equality and taking a lim sup of ([59]) yields limsup^^^^E [^(t)] < e. This holds for all e > 0, and so 
limsupi^^ E [X{t)] < 0. Similarly, it can be shown that liminft^oo E [X{t)] > 0. Thus, limt^oo E [X{t)] = 0. □ 
Recall that convergence with probability 1 is stronger than convergence in probability, and so the above result also 
holds if limj_j.oo X{t) = x with probability 1. Theorem 2 can be applied to the case when lim/^_j.oo ^ 'l2k=o ^i^] ~ 
e with probability 1, for some finite constant e, and when there is a constant C such that E [e[A;]^] < C for all k. 
Indeed, one can define X{K) = ^ '^k=o ^i^] ^ ^ 3, . . .} and use the Cauchy-Schwartz inequality to 
show E [X{Kf] < C for all K G {1, 2, 3, . . .}, and so lim^^^oo Ef=o^ ^ [eM = e. 

Appendix B — The Decision Rule for Linear Fractional Programming 

This appendix proves optimality of the decision rule for online linear fractional programming given in Exercise 
[6] Specifically, suppose we have real numbers (po, (pi, . . . , (pM, 3. positive real number bo > 0, and non-negative real 
numbers 61, ... , bu- The goal is to solve the following problem: 

Minimize: t"lS^^?'"' 
Subject to: < x^ < 1 Vi G {1, . . . , M} 
To this end, define disjoint sets Z and J' that partition the indices {!,..., M} as follows: 

X4{iG {!,..., M}|6i = 0} , j4{iG {!,..., M}|6,>0} 
Then the objective function can be written: 

bo + EZi biXi bo + bjXj bo + Ej&j bjXj 

The denominator is always positive, and so regardless of the Xj values for j G the first term on the right-hand- 
side above is minimized by choosing Xj for all i ^ I such that Xi = if (pi > 0, and Xi = 1 if (pi < 0. Let x* for 
z G X denote these decisions. It remains only to choose optimal Xj values for j G J^. Define ^o=4'o + J2i&x'Pi^i- 
We want to solve the following: 

Minimize: rtP^^-^fC' (61) 
Subject to: < xj < 1 \fj e J (62) 



Define val* as the minimum value for the objective function in the problem ([6T|)-([62|). 

Lemma 4: The following values {x*)j<^j solve the problem (|6T])-(|62l): For each j G J, we have: 

r if > val* 

i \ 1 if (Pj/bj < val* ^ ' 

Proof: Let {xj)j^j be an optimal solution to (61 l-(|62l), so that < Xj < 1 for all j ^ J and: 



val* 



70 + Ejej ' 



bo + EjejbjXj 
Define c and d by: 

c=7o + ^ (l^jXj , d=bo + ^ hj 
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val 



so that val* = c/d. Now take any k £ J'. It suffices so show we can change to x^, as defined in ([63]), without 
changing the ratio from its value val*. Indeed, if this is true, then we can sequentially change each component Xj 
to X* without changing the ratio, and so {x*)j;^j is also an optimal solution. 

To this end, define val as the corresponding ratio with Xk replaced with x*^. We show that val = val*. Clearly 
val > val* by definition of val* as the minimum ratio. To show the opposite inequality, define 6^ = x^. — Xk, and 
note that: 

c + (l)kSl 
d + bkSl 

C + (pk^l c 

2 d + bkdl d 
c cd + 4>k5*^d — cd — cbi^Sl 
d^ d{d + bkSl) 

val* + 61 [(Pk/bk - val*] -r^^ (64) 



Note that bk/ {d + bk5*^) > 0, and hence the second term on the right-hand-side of (64 1 is non-positive if 5l[4>k/bk — 
val*] < 0. In the case (j)k/bk > val*, then by ( [63] ) we have xl = 0. Thus, §1 <0 and (5^[(/>fc/6fc — val*] < 0. Thus, 
the second term in the right-hand-side of (]64]) is non-positive and so val < val*. In the opposite case (pk/bk < val*, 
then xl = I and so 51 > 0, 6l[(l)k/bk — val*] < 0. Thus, the second term on the right-hand-side of ([64]) is again 
non-positive, so that val < val*. □ 
Now rank order the indices j £ J from smallest to largest value of (pj/bj, breaking ties arbitrarily. The solution 
of Lemma |4] has the form where = 1 for the first n indices in this rank order, and x* = for the remaining 
indices, for some value n G {0, 1, . . . , \ J^\} (where \ J'\ is the size of set J'). Define vain as the value of the ratio 
when Xj = 1 for the first n indices in the rank order, and Xj = for the remaining indices in J'. Start with 
valQ = 7o/6o- According to the rank ordering of J', successively change Xj from to 1 if it strictly decreases 
the ratio, computing vali, val2, and so on, until we reach the first index that does not improve the ratio. By an 



argument similar to (64i, it is easy to see that this can only occur at an index j such that is greater than 

or equal to the current ratio, which means it is also greater than or equal to val* (since by definition, val* is less 
than or equal to any achievable ratio). Thus, all indices i £ J that come after j in the rank ordering also have 



(t^i/bi > val*, and this greedy approach has thus arrived at the solution (63 1. 
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